Psychological Review

Extensive research in the behavioural sciences has addressed peoples ability to learn stationary probabilities, which stay constant over time, but only recently have there been attempts to model the cognitive processes whereby people learn - and track - non-stationary probabilities. In this context, the old debate on whether learning occurs by gradual formation of associations or by occasional shifts between hypotheses representing beliefs about distal states of the world has resurfaced. Gallistel et al. (2014) pitched the two theories against each other in a non-stationary probability learning task. They concluded that various qualitative patterns in their data were incompatible with trial-by-trial associative learning and could only be explained by a hypothesis-testing model. Here, we contest that claim and demonstrate that it was premature. First, we argue that their experimental paradigm consisted of two distinct tasks: probability tracking (an estimation task) and change detection (a decision-making task). Next, we present a model that uses the (associative) delta learning rule for the probability tracking task and bounded evidence accumulation for the change-detection task. We find that this combination of two highly established theories accounts well for all qualitative phenomena and outperforms the alternative model proposed by Gallistel et al. in a quantitative model comparison. In the spirit of cumulative science, we conclude that current experimental data on human learning of non-stationary probabilities can be explained as a combination of associative learning and bounded evidence accumulation and does not require a new model.

The drift-diffusion model (DDM) is widely used and broadly accepted for its ability to account for binary choices (in both the perceptual and preferential domains) and response times (RT), as a function of the stimulus or the choice alternative (or option) values. The DDM is built on an evidence accumulation-to-bound concept, where, in the value domain, a decision maker repeatedly samples the mental representations of the values of the available options until satisfied that there is enough evidence (or support) in favor of one option over the other. As the signals that drive the evidence are derived from value estimates that are not known with certainty, repeated sequential samples are necessary to average out noise. The classic DDM does not allow for different options to have different levels of precision in their value representations. However, recent studies have shown that decision makers often report levels of certainty regarding value estimates that vary across choice options. There is therefore a need to extend the DDM to include an option-specific value certainty component. We present several such DDM extensions and validate them against empirical data from four previous studies. The data support best a DDM version in which the drift of the accumulation is based on a sort of signal-to-noise ratio of value for each option (rather than a mere accumulation of samples from the corresponding value distributions). This DDM variant accounts for the impact of value certainty on both choice consistency and response time present in the empirical data.

Sequential sampling models of choice, such as the drift-diffusion model (DDM), are frequently fit to empirical data to account for a variety of effects related to accuracy/consistency, response time (RT), and sometimes confidence. However, no model in this class can account for the phenomenon known as choice-induced preference change, where decision makers tend to rate options higher after they choose them and lower after they reject them. Studies have reported choice-induced preference change for many decades, and the principal findings are robust. The resulting spreading of alternatives (SoA) in terms of their subjective value ratings is incompatible with traditional sequential sampling models, which consider the rated values of the options to be stationary throughout choice deliberation. Here, we propose an extension of the basic DDM that allows the drift rate to vary across deliberation time depending on which attributes are attended to at which points in time. Critically, the model assumes that choice deliberation commences based only on the more salient attributes of the options, and that additional attributes eventually come into consideration when the decision cannot be resolved based on the initial attributes alone. We show that this model can account for SoA (in addition to choice consistency and RT), as well as all previously reported relationships between SoA and choice difficulty, attribute disparity, and RT.

Associative learning models have traditionally simplified contingency learning by relying on binary classification of cues and outcomes, such as administering a medical treatment (or not) and observing whether the patient recovered (or not). While successful in capturing fundamental learning phenomena across human and animal studies, these models are not capable of representing variability in human experiences that are common in many real-world contexts. Indeed, where variation in outcome magnitude exists (e.g., severity of illness in a medical scenario), this class of models, at best, approximate the outcome mean with no ability to represent the underlying distribution of values. In this paper, we introduce one approach to incorporating a distributed architecture into a prediction error learning model that tracks the contingency between cues and dimensional outcomes. Our Distributed Model allows associative links to form between the cue and outcome nodes that provide distributed representation depending on the magnitude of the outcome, thus enabling learning that extends beyond approximating the mean. Comparing the Distributed Model against a Simple Delta Model across four contingency learning experiments, we found that the Distributed Model provides significantly better fit to empirical data in virtually all participants. These findings suggest human learners rely on a means of encoding outcomes that preserves the continuous nature of experienced events, advancing our understanding of causal inference in complex environments. Author SummaryWhen we learn about cause and effect in everyday life--such as whether a medicine helps recovery from illness--we experience outcomes that vary in degree rather than simply happening or not happening. Traditional models of how humans and animals learn have largely focused on these all-or-nothing scenarios, essentially tracking the average value when outcomes are dimensional. We developed a model that extends on simple error-correction models to represent how people learn about relationships between cues and outcomes that can take on a range of values. Instead of just tracking the average, our Distributed Model captures the full spectrum of possible outcomes and their frequencies. We tested this model against a conventional single point-estimate approach across four experiments and found that our Distributed Model better matched how people make predictions in nearly every case. Our findings suggest that a relatively simple adjustment to conventional prediction-error learning algorithms that allows representation of outcome magnitudes provide a powerful way to capture the information that we preserve when we learn about variable outcomes. This has important implications for understanding how people make predictions and decisions in real-world situations where outcomes naturally vary, from medical treatments to environmental changes.

The notion of a response criterion is ubiquitous in psychology, yet its cognitive and neural underpinnings remain poorly understood. To address this shortcoming, three computational models that capture different hypotheses about criterial learning were developed and tested. The time-dependent drift model assumes the criterion is stored in working memory and that its value drifts over time. The delay-sensitive learning model assumes that the magnitude of criterial learning is temporally discounted by feedback delay. The reinforcement-learning model assumes that criterial learning emerges from stimulus-response association learning without an explicit representation of the criterion, with learning rate also temporally discounted by feedback delay. The performance of these models was investigated under varying feedback delay and intertrial interval (ITI) durations. The time-dependent drift model predicted that long ITIs and feedback delays both impair criterial learning. In contrast, the delay-sensitive and reinforcement-learning models predicted impairments only with feedback delays. Two behavioral experiments, which tested these predictions, showed that human criterial learning is impaired by delayed feedback but not by long ITIs. These results support the delay-sensitive and reinforcement-learning models, and suggest that even in tasks that appear to rely on explicit, rule-based reasoning, criterial learning may have strong associative underpinnings.

Many studies in perception and in the working memory literature demonstrate that human observers systematically deviate from the truth when estimating the features of one item in the presence of another. Such inter-item or contextual biases are well established but lack a coherent explanation at the computational level. Here, I propose a novel normative model showing that such biases exist for any observer striving for optimality when trying to infer the features of multiple similar objects from a mixture of sensory observations. The demixing model predicts that bias strength and direction would vary as a function of the amount of sensory noise and the similarity between items. Crucially, these biases exist not because of the prior knowledge in any form, but simply because the biased solutions to this inference problem are more probable than unbiased ones, counter to the common intuition. The model makes novel predictions about the effect of discriminability along the dimension used to select the item to report (e.g., spatial location) and the relative amount of sensory noise. Although the model is consistent with previously reported data from human observers, more carefully controlled studies are needed for a stringent test of its predictions. The strongest point of the demixing model, however, is that it shows that interitem biases are inevitable when observers lack perfect knowledge of which stimuli caused which sensory observations, which is, arguably, always the case.

In a recent bioRxiv preprint, Ott et al. argue that sensitivities to sunk costs that have been reported in two serial foraging tasks (the Restaurant Row task in mice and rats, and the Web-Surf task in humans) may be due to simple consequences of the way that subjects perform these tasks and not due to an actual sensitivity to sunk costs. However, several variants of these tasks have been studied, in which the sensitivity to sunk costs changes. In order to test the Ott et al. model against these experimental observations, we simulated the model under these additional experimental conditions. We find that it is incompatible with the actual data. While we applaud the simplicity of the Ott et al. model, we must reject it as an explanation for the observed sensitivity to sunk costs seen in these tasks. We thus conclude that the alternative explanation - that mice, rats, and humans are sensitive to actual sunk costs in these tasks - is a better explanation for the data.

Interpreting sensory prediction errors can be challenging in volatile environments because they can be caused by stochastic noise or by outdated predictions. Noisy signals should be integrated with prior beliefs to improve precision, but the two should be segregated when environmental changes render prior beliefs irrelevant. Bayesian causal inference provides a statistically optimal solution to deal with uncertainty about the causes of prediction errors. However, the method quickly becomes memory intensive and computationally intractable when applied sequentially. Here, we systematically evaluate the predictive performance of Bayesian causal inference for perceptual decisions in a spatial prediction task based on noisy audiovisual sequences with occasional changepoints. We elucidate the simplifying assumptions of a previously proposed reduced Bayesian observer model, and we compare it to an extensive set of models based on alternative simplification strategies. Model-free analyses revealed the hallmarks of Bayesian causal inference: participants seem to have integrated sensory evidence with prior beliefs to improve accuracy when prediction errors were small, but prior influence decreased gradually as prediction errors increased, signalling probable irrelevance of the priors due to changepoints. Model comparison results indicated that participants computed probability-weighted averages over the causal options (noise or changepoint), akin to the reduced Bayesian observer model. However, participants reliance on prior beliefs was systematically smaller than expected, and this was best explained by individually fitting lower-than-optimal parameters for the a-priori probability of prior relevance. We conclude that perceptual decision makers utilize priors flexibly to the extent that they are deemed relevant, though also conservatively with a lower tendency to bind than ideal observers. Simplified consecutive Bayesian causal inference predicts key characteristics of belief updating in changepoint environments and forms a suitable foundation for modelling dynamic perception in a changing world.

Stimulus-selective stopping extends the standard stop signal task by occasionally presenting an ignore signal instead of a stop signal, in which case participants are instructed to continue responding to the go signal. Here we present several model classes that are based on the idea that responses observed under an ignore signal are the result of a probabilistic mixture from the processing distributions of the go and the ignore signal. Earlier work ensures that the mixture hypothesis is statistically testable. We derive quantitative predictions and parameter estimation for model classes that differ in the way the mixture is introduced. The results are illustrated with an application to a published dataset for stimulus-selective stopping.

Human decision behavior entails a graded awareness of its certainty, known as a feeling of confidence. Until now, considerable interest has been paid to behavioral and computational dissociations of decision and confidence, which has raised an urgent need for measurement frameworks that can quantify the efficiency of confidence rating relative to decision accuracy (metacognitive efficiency). As a unique addition to such frameworks, we have developed a new signal detection theory paradigm utilizing the generalized gaussian distribution (GGSDT). This framework evaluates the observers internal standard deviation ratio and metacognitive efficiency through the scale and shape parameters respectively. The shape parameter quantifies the kurtosis of internal distributions and can practically be understood in reference to the proportion of the gaussian ideal observers confidence being disrupted with random guessing (metacognitive lapse rate). This interpretation holds largely irrespective of the contaminating effects of decision accuracy or operating characteristic asymmetry. Thus, the GGSDT enables hitherto unexplored research protocols (e.g., direct comparison of yes/no versus forced-choice metacognitive efficiency), expected to find applications in various fields of behavioral science. This paper provides a detailed walkthrough of the GGSDT analysis with an accompanying R package (ggsdt).

1What are the contents of working memory? In both behavioral and neural computational models, a working memory representation is typically described by a single number, namely a point estimate of a stimulus. Here, we asked if people also maintain the uncertainty associated with a memory, and if people use this uncertainty in subsequent decisions. We collected data in a two-condition orientation change detection task; while both conditions measured whether people used memory uncertainty, only one required maintaining it. For each condition, we compared an optimal Bayesian observer model, in which the observer uses an accurate representation of uncertainty in their decision, to one in which the observer does not. We find that this "Use Uncertainty" model fits better for all participants in both conditions. In the first condition, this result suggests that people use uncertainty optimally in a working memory task when that uncertainty information is available at the time of decision, confirming earlier results. Critically, the results of the second condition suggest that this uncertainty information was maintained in working memory. We test model variants and find that our conclusions do not depend on our assumptions about the observers encoding process, inference process, or decision rule. Our results provide evidence that people have uncertainty that reflects their memory precision on an item-specific level, maintain this information over a working memory delay, and use it implicitly in a way consistent with an optimal observer. These results challenge existing computational models of working memory to update their frameworks to represent uncertainty.

We present the Spreading Activation and Memory PLasticity Model (SAMPL), a computational model of how memory retrieval changes memories. SAMPL restructures memory networks as a function of spreading activation and plasticity. Memory networks are represented as graphs of items in which edge weights capture the strength of association between items. When an item is retrieved, activation spreads across nodes depending on edge weights and the strength of initial activation. A non-monotonic plasticity rule, in turn, updates edge weights following activation. SAMPL simulates human memory phenomena across a number of experiments including retrieval induced forgetting, context-based memory enhancement, and memory synchronization in conversational networks. Our results have implications for theorizing memory disorders such as PTSD and designing computationally assisted conversational therapy.

Whether we are making life-or-death decisions or thinking about the best way to phrase an email, counterfactual emotions including regret and disappointment play an ever-present role in how we make decisions. Functional theories of counterfactual thinking suggest that the experience and future expectation of counterfactual emotions should promote goal-oriented behavioral change. Although many studies find empirical support for such functional theories, the generative cognitive mechanisms through which counterfactual thinking facilitates changes in behavior are underexplored. Here, we develop generative models of risky decision-making that extend regret and disappointment theory to experience-based tasks, which we use to examine how people incorporate counterfactual information into their decisions across time. Further, we use computer-vision to detect positive and negative affect (valence) intensity from participants faces in response to feedback, which we use to explore how experienced emotion may correspond to cognitive mechanisms of learning, outcome valuation, or exploration/exploitation--any of which could result in functional changes in behavior. Using hierarchical Bayesian modeling and Bayesian model comparison methods, we found that a model assuming: (1) people learn to explicitly represent and subjectively weight counterfactual outcomes with increasing experience, and (2) people update their counterfactual expectations more rapidly as they experience increasingly intense negative affect best characterized empirical data. Our findings support functional accounts of regret and disappointment and demonstrate the potential for generative modeling and model-based facial expression analysis to enhance our understanding of cognition-emotion interactions.

A mathematical idealization of the way neural populations encode sensory information has been found to provide a parsimonious account of errors made by human observers on perceptual and short-term memory tasks. This includes the effects of set size and flexible prioritization of items within a set (Bays, 2014), the frequency and identity of \"swap\" or misbinding errors (Schneegans & Bays, 2017), subjective judgments of confidence (Bays, 2016; van den Berg et al., 2017), and biases and variation in precision due to serial dependent and stimulus-specific effects (Bliss et al., 2017; Taylor & Bays, 2018). A superficially quite different account of short-term recall has recently been proposed in work by Schurgin et al. (2018), who argue that taking into account the differences between physical and perceptual distance in a feature space reduces recall to a classical signal detection problem. Here I document a remarkable similarity between the two models, demonstrating a favourable convergence of neural- and cognitive-level accounts of working memory.

Reinforcement learning models describe how agents learn about the world (value learning), and how they interact with their environment based on the learned information (decision policy). As in any optimization problem, it is important to set the process hyperparameters, a process which also is thought to be learned (meta-learning). Here, we test a key prediction of meta-learning frameworks, namely that there exist one or more meta-signals that govern hyperparameter setting. Specifically, we test whether decision confidence, in a context of varying outcome variability, informs hyperparameter setting. Participants performed a 2-armed bandit task with confidence ratings. Model comparison shows that confidence and outcome variability are differentially involved in hyperparameter setting. A high level of confidence in the previous choice decreased hyperparameter setting of decision noise on the current trial: when a trial was made with low confidence, the choice on the next trial tended to be more explorative (i.e. high decision noise). Outcome variability influenced another hyperparameter, the learning rate for positive prediction errors (thus affecting value learning). Both strategies are rational approaches that maximize earnings at different temporal loci: the modulation by confidence causes more frequent exploration early after a change point, the modulation by outcome variability is advantageous late after a change point. Finally, we show that (reported) confidence in value-based choices reflects the action value of the chosen option (irrespective of the unchosen value). In sum, decision confidence and outcome variability reflect distinct signals that optimally guide the setting of hyperparameters in decision policy and value learning, respectively.

We view observational causal induction as a statistical independence test under rarity assumption. This paper complements the two-stage theory of causal induction proposed by Hattori and Oaksford (2007) with a computational analysis. We show that their dual-factor heuristic (DFH) model has a rational account as the square root of the index of (non-)independence under extreme rarity assumption, contrary to the criticism that the DFH model is non-normative (e.g., Lu et al., 2008). We introduce a model that considers the proportion of assumed-to-be rare instances (pARIs), which is the probability of biconditionals (according to several theories of compound conditionals) and can be seen as a simplified version of the DFH model. While being a single conditional probability, pARIs approximates the non-independence measure, the square of DFH. In reproducing the meta-analysis in Hattori and Oaksford (2007), we confirm that pARIs and DFH have the same level of descriptive adequacy, and that the two models have the highest fit among more than 40 models. Then, we critically examine the computer simulations which were central to the rational analysis in Hattori and Oaksford (2007). We point out two problems in their simluations: samples in some of the simulations being restricted to generative ones, and in-definite values of models because of the small samples. In the light of especially the latter problem of definability, pARIs shows higher applicability.

A key concept in reinforcement learning (RL) is that of a state space. A state space is an abstract representation of the world using which statistical relations in the world can be described. The simplest form of RL, model free RL, is widely applied to explain animal behavior in numerous neuroscientific studies. More complex RL versions assume that animals build and store an explicit model of the world in memory. To apply these approaches to explain animal behavior, typical neuroscientific RL models make assumptions about the underlying state space formed by animals, especially regarding the representation of time. Here, we explicitly list these assumptions and show that they have several problematic implications. We propose a solution for these problems by using a continuous time Markov renewal process model of the state space. We hope that our explicit treatment results in a serious consideration of these issues when applying RL models to real animals.

Animals continuously infer latent properties of the world from noisy and changing observations. Complex approaches to this challenge such as Bayesian inference are accurate but cognitively demanding, requiring extensive working memory and adaptive learning. Simple strategies such as always using a prior bias or following the last observation are easy to implement but may be less accurate. What is the appropriate balance between complexity and accuracy? We construct a hierarchy of strategies that vary in complexity between these limits and find a power law of diminishing returns: increasing complexity gives progressively smaller gains in accuracy. Moreover, the rate at which the gain decrements depends systematically on the statistical uncertainty in the world, such that complex strategies do not provide substantial benefits over simple ones when uncertainty is too high or too low. In between, when the world is neither too predictable nor too unpredictable, there is a complexity dividend.

In games of incomplete information individual players make decisions facing a combination of structural uncertainty about the underlying parameters of the environment, and strategic uncertainty about the actions undertaken by their partners. How well are human actors able to cope with these uncertainties, and what models best describe their learning in such environments? We use a double auction task with different competitive and informational environments to characterize learning abilities of the single human participants (buyers) in a range of adaptive learning models covering reinforcement learning, directional learning and belief learning. Results show that real behaviour is best described using simple models of directional learning type with minimal knowledge assumptions about information efficiency of prices. This behavior is consistent with bounded rationality and risk aversion: human subjects try to maximize their chance for transaction, and do so using the simplest learning rule.

Humans have been shown to adapt their movements when a sudden or gradual change to the dynamics of the environment are introduced, a phenomenon called motor adaptation. If the change is reverted, the adaptation is also quickly reverted. Humans are also able to adapt to multiple changes in dynamics presented separately, and to be able to switch between adapted movements on the fly. Such switching relies on contextual information which is often noisy or misleading, affecting the switch between known adaptations. Recently, computational models for motor adaptation and context inference have been introduced, which contain components for context inference and Bayesian motor adaptation. These models were used to show the effects of context inference on learning rates across different experiments. We expanded on these works by using a simplified version of the recently-introduced COIN model to show that the effects of context inference on motor adaptation and control go even further than previously shown. Here, we used this model to simulate classical motor adaptation experiments from previous works and showed that context inference, and how it is affected by the presence and reliability of feedback, effect a host of behavioral phenomena that had so far required multiple hypothesized mechanisms, lacking a unified explanation. Concretely, we show that the reliability of direct contextual information, as well as noisy sensory feedback, typical of many experiments, effect measurable changes in switching-task behavior, as well as in action selection, that stem directly from probabilistic context inference.